Information Fusion for Visual Reference Resolution in Dynamic Situated Dialogue
نویسندگان
چکیده
Human-Robot Interaction (HRI) invariably involves dialogue about objects in the environment in which the agents are situated. The paper focuses on the issue of resolving discourse references to such visual objects. The paper addresses the problem using strategies for intra-modal fusion (identifying that different occurrences concern the same object), and inter-modal fusion, (relating object references across different modalities). Core to these strategies are sensorimotoric coordination, and ontology-based mediation between content in different modalities. The approach has been fully implemented, and is illustrated with several working examples.
منابع مشابه
Visual Salience and Reference Resolution in Situated Dialogues: A Corpus-based Evaluation
Dialogues between humans and robots are necessarily situated. Exophoric references to objects in the shared visual context are very frequent in situated dialogues, for example when a human is verbally guiding a tele-operated mobile robot. We present an approach to automatically resolving exophoric referring expressions in a situated dialogue based on the visual salience of possible referents. W...
متن کاملIncorporating Extra-Linguistic Information into Reference Resolution in Collaborative Task Dialogue
This paper proposes an approach to reference resolution in situated dialogues by exploiting extra-linguistic information. Recently, investigations of referential behaviours involved in situations in the real world have received increasing attention by researchers (Di Eugenio et al., 2000; Byron, 2005; van Deemter, 2007; Spanger et al., 2009). In order to create an accurate reference resolution ...
متن کاملReference Resolution in Situated Dialogue with Learned Semantics
Understanding situated dialogue requires identifying referents in the environment to which the dialogue participants refer. This reference resolution problem, often in a complex environment with high ambiguity, is very challenging. We propose an approach that addresses those challenges by combining learned semantic structure of referring expressions with dialogue history into a ranking-based mo...
متن کاملTowards Situated Open World Reference Resolution
Natural language dialogue provides the opportunity for truly natural human-robot interaction. A robot participating in natural language dialogue must identify or create new representations for referenced entities if it is to discuss, reason about, or perform actions involving that entity, a capability known as reference resolution. In previous work we presented algorithms for resolving referenc...
متن کاملMulti-modal Reference Resolution in Situated Dialogue by Integrating Linguistic and Extra-Linguistic Clues
This paper focuses on examining the effect of extra-linguistic information, such as eye gaze, integrated with linguistic information on multi-modal reference resolution. In our evaluation, we employ eye gaze information together with other linguistic factors in machine learning, while in prior work such as Kelleher (2006) and Prasov and Chai (2008) the incorporation of eye gaze and linguistic c...
متن کامل